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1  Project  Summary 


Traditional  system  identification  addresses  the  task  of  inferring  a  mathematical  model  of  a 
system  from  observations  of  that  system.  A  controls  engineer  might  perform  this  task,  in  its 
most  basic  form,  by  choosing  a  power  series  and  matching  its  coefficients,  via  some  sort  of 
regression,  to  a  numerical  time  series  from  a  sensor.  The  goal  of  this  project  is  the  development 
of  a  computer  program,  called  pret,  that  automates  the  entire  system  identification  process 
by  building  an  artificial  intelligence  (AI)  layer  around  a  set  of  traditional  system  identification 
techniques.  Ihis  AI  layer  executes  many  of  the  high-level  parts  of  the  identification  process 
that  are  normally  performed  by  a  human  expert,  using  qualitative,  symbolic,  and  geometric 
reasoning  to  perform  structural  identitication,  intelligently  guiding  the  search  to  the  proper  type 
and  form  of  equation.  Unknown  coefficients  are  then  found  with  special,  Al-adapted  parameter 
ideiitiiication  techniques. 

PRET  works  with  ordinary  differential  equation  (ODE)  models,  linear  or  nonlinear,  in  one 
variable  or  many.  Its  implementation  combines  traditional  numerical  analysis  methods,  such 
as  nonlinear  regression,  with  AI  techniques  like  logic  programming,  symbolic  and  geometric 
reasoning,  and  constraint  propagation.  The  inputs  consist  of  specific  information  about  an 
individual  system,  in  three  forms: 

•  the  user’s  hypotheses  about  the  physics  involved 

•  observations,  interpreted  and  described  by  the  user,  symbolically  or  graphically,  in  varying 
formats  and  degrees  of  precision 

•  physical  measurements  made  directly  and  automatically  on  the  system 

To  construct  an  ODE  model  from  this  information,  pret  combines  powerful  mathematical  for¬ 
malisms,  such  as  the  link  between  the  divergence  of  an  ODE  and  the  friction  of  the  system  that 
it  describes,  with  domain-specific  notions  —  such  as  force  balances  in  mechanical  systems  —  to 
allow  the  types  of  custom-generated  approximations  that  are  lacking  in  existing  AI  modeling 
programs.  Two  different  sets  of  rules,  both  of  which  may  easily  be  changed  or  augmented  by 
the  user,  play  critical  roles  in  the  model-building  task.  Domain-specific  rules  are  used  in  the 
generate  phase  to  combine  hypotheses  into  models  —  a  nontrivial  task  in  a  system  with  more 
than  one  degree  of  freedom,  or  a  system  in  which  physical  effects  couple  to  one  another  —  while 
general  rules  about  ODE  properties  are  used  by  a  custom  first-order  logic  system  to  rule  out 
models  that  conflict  with  the  observations. 


2  FY9 7  Research  Accomplishments 

•  PRET  efficiently  and  effectively  solves  a  class  of  global  optimization  problems  involved  in 
the  process  of  parameter  estimation.  Specifically,  qualitative  reasoning  (QR)  techniques 
are  used  to  derive  starting  values  for  a  nonlinear  least-squares  solver  call.  These  good 
initial  approximations  guide  the  NLS  solver  away  from  local  extrema  in  a  complicated 
nonlinear  regression  landscape.  QR  techniques  are  also  used  to  compute  cutoff  frequencies 
for  digital  filters  in  order  to  identify  and  remove  noise. 
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Our  paper  on  this  topic  was  selected  —  along  with  with  seven  others,  out  of  a  total  of  76 
—  for  a  presentation  of  the  highlights  of  the  1996  and  1997  QR  workshops  that  is  to  be 
given  at  IJCAI  1997. 

•  PRET  constructs  models  in  a  new^  domain,  viscoelastics,  which  has  been  the  focus  of  state- 
of-the-art  AI  modeling  programs  by  other  QR  groups.  It  was  very  easy  for  us  to  add  this 
new  domain  because  of  the  flexibility  and  extensibility  designed  into  PRET’s  framework. 
This  ease  —  and  the  comparisons  to  other  work  that  the  new  domain  implementation 
allowed  —  proved  positive  and  satisfying. 

•  PRET  automatically  synthesizes  model  fragments  from  scratch  using  power- series  tech¬ 
niques.  This  technique  is  ubiquitous  in  engineering,  but  nonexistent  in  AI  modeling  tools. 
It  is  not  only  powerful,  but  was  also  very  easy  to  implement:  for  instance,  a  one-  or 
two-term  expansion  in  the  first  few  derivatives  of  a  state  variable  yield  every  form  of  force 
term  found  in  freshman  physics  textbooks. 

•  PRET  autonomously  processes  sensor  data  to  construct  observations  of  a  target  system. 
This  involved  solving  part  of  control  theory’s  observer  problem:  inferring  internal  system 
dynamics  from  incomplete  output  sensor  data.  We  accomplish  this  using  delay-coordinate 
embedding  and  then  perform  feature  extraction  on  the  resulting  reconstructed  dynamics 
with  some  simple  geometric  reasoning/computer  vision  techniques. 

Our  paper  on  these  results,  which  was  presented  at  IDA97  in  London,  was  judged  to  be 
among  the  best  papers  of  that  conference. 

•  PRET  now  uses  backward  (SLD)  reasoning^.  Backward  reasoning  is  goal-directed,  allowing 
invalid  models  to  be  ruled  out  more  quickly,  and  it  allows  for  smooth  integration  of  the 
truth-maintenance  module. 

•  We  implemented  a  declarative  meta-level  control  language  with  which  the  user  can  dy¬ 
namically  control  the  order  in  which  inference  rules  are  applied.  The  reason  for  this  is  that 
effective  rule  order  choice  often  depends  on  current  knowledge,  and  static  rule  ordering 
does  not  allow  for  the  type  of  flexible  control  flow  that  exploits  that  knowledge. 

•  Finally,  and  perhaps  most  interesting  of  aU,  we  have  successfully  used  PRET  on  a  real- 
world  example  —  a  radio-controlled  car  being  used  in  a  robotics  system  at  the  University 
of  British  Columbia,  pret’s  model  was  actually  better  than  the  one  constructed  by 
the  project  analyst  in  several  senses,  and  the  discrepancies  helped  the  analyst  refine  his 
mental  model  of  the  system.  More  information  on  this  example  is  included  in  section  4; 
powerpoint  graphics  will  be  enclosed  in  a  separate  email  message. 

3  FY98  Research  Goals 

While  the  items  in  the  previous  section  represent  significant  successes  —  and  large  portions 
of  the  “to  do”  list  from  our  1996  Progress  Report  —  the  ultimate  goal  of  this  line  of  research 

^Previously,  PRET  could  only  solve  classical  mechanics  problems. 

^instead  of  breadth-first  forward  reasoning 
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is  a  tool  that  can  construct  models  of  high-dimensional  black-box  systems,  drawn  from  any 
domain  that  admits  ODE  models,  using  only  information  that  is  observable  from  the  ports  of 
those  systems.  In  order  for  PRET  to  attain  the  next  level  of  this  ambitious  plan,  a  variety  of 
significant  improvements  —  the  subject  of  the  work  to  be  performed  under  the  remaining  years 
of  this  contract  —  are  necessary.  These  fall  in  four  general  areas,  as  listed  below: 

1.  Expand  P RET ’s  capability  to  handle  more-complex  applications  in  more  domains  (elec¬ 
tronics,  chemical  kinetics,  etc.),  more  efficiently,  using  a  richer  rule  set. 

2.  Use  truth  maintenance  techniques  to  efficiently  maintain  partial  proof  trees  that  document 
the  reasoning  that  has  been  performed.  These  proof  trees  avoid  duplication  of  effort  and 
will  also  be  used  to  guide  the  generation  of  successor  candidate  models. 

3.  Develop  and  incorporate  analysis  and  control  techniques  to  actively  and  realistically  ex¬ 
ploit  sensors  and  actuators  to  allow  a  true  input-output  approach  to  modeling,  another 
idea  that  is  relatively  common  in  system  identification  but  rare  in  AI. 

4.  Implement  a  graphical  user  interface  to  make  the  program  easier  for  domain  experts  to 
use. 


As  mentioned  above,  we  have  made  significant  inroads  on  items  1-3,  and  we  have  successfully 
finished  one  of  last  year’s  action  items  —  the  power  series  techniques  for  automatic  hypothesis 
generation.  Item  4  is  a  new  addition;  our  current  interface  makes  demos  rather  painful.  Specific 
subgoals  for  each  item  above  are: 

•  Item  1: 

-  We  will  continue  adding  domains  to  PRET.  Electronics  is  currently  first  on  the  list; 
implementing  this  domain  is  nontrivial,  as  it  will  require  us  to  rework  and  probably 
augment  the  syntax  with  which  coordinates  and  connections  are  described  to  the 
program.  (Bradley,  Easley,  Hogan,  Stolle) 

—  We  will  add  rules  as  we  test  PRET  on  more  examples  in  different  domains.  (Bradley, 
Easley,  Hogan,  Stolle) 

-  We  will  begin  collaborating  with  a  group  at  MIT,  headed  by  David  Trumper  in 
EECS,  that  is  performing  system  identification  on  a  commercial  eyeglass  lens-cutting 
machine,  and  attempt  to  reproduce  their  results  automatically.  (Bradley,  Easley, 
Hogan,  Stolle) 

•  Item  2: 

-  We  will  finish  the  truth-maintenance  system  that  allows  knowledge  reuse  and  pro¬ 
vides  explanations  for  the  failure  of  models.  (Hogan,  Stolle) 

—  We  will  finish  the  constraint  propagation  algorithm  that  will  process  constraint  ex¬ 
pressions  on  parameter  values  and  state  variables  —  for  the  purposes  of  (1)  setting 
up  the  parameter  estimation  call  (2)  informing  users  about  possible  parameter  values 
or  (3)  if  the  constraints  cannot  be  satisfied,  discarding  the  model.  (Hogan,  Stolle) 
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-  Stolle  will  defend  his  Ph.D.  thesis. 


•  Item  3: 


-  We  will  finish  setting  up  the  haxdware  and  software  for  the  data  acquisition  and 
control  system  and  complete  the  interface  that  allows  pret  to  interact  with  this 
equipment.  (Easley;  this  was  put  on  hold  in  FY97  by  hardware  problems.) 

-  We  will  build  and  test  another  example  system  —  probably  a  spring-mass  oscillator 
on  an  air  track.  (Easley) 

—  We  will  continue  working  out  the  syntax,  semantics,  and  representation  of  the  in¬ 
formation  PRET  receives  about  the  sensors  and  sends  to  the  actuators.  (Bradley, 
Easley) 

-  We  will  begin  to  automate  pret’s  output  interactions.  This  involves  automating 
much  of  what  control  theorists  call  controllability  and  reachability.  (Bradley,  Easley) 

-  Easley  will  defend  his  Ph.D.  proposal. 

•  Item  4: 

—  We  build  a  GUI  and  test  it  out,  hopefully  using  someone  from  the  MIT  lens-cutting 
machine  project  as  a  test  case.  (Hogan) 


4  Technical  Transitions 


As  of  last  year,  members  of  our  group  had  verified  pret’s  performance  on  some  of  the  textbook 
examples  used  in  the  graduate  system  identification  course  offered  by  the  Aerospace  Engineering 
Department  at  the  University  of  Colorado.  Since  then,  PRET  has  been  developed  to  the  point 
where  demos  to  outside  people  have  become  routine,  which  has  elicited  a  variety  of  useful 
suggestions  and  directions  (e.g.,  the  GUI). 

We  have  also  used  PRET  on  some  real-world  problems,  including  a  radio-controlled  car  being 
used  in  a  robotics  system  at  the  University  of  British  Columbia.  The  input  consisted  of  data 
files  and  email  messages  containing  hypotheses  about  the  physics  (e.g.,  “0  =  pu”:  the  heading 
changes  linearly  with  velocity)  and  qualitative  observations  of  the  R/C  car  (e.g.,  “it  pulls  to 
the  left”).  The  goal  was  to  duplicate  the  model  found  by  a  human  analyst,  but  pret’s  model 
was  actually  better,  in  several  senses.  It  was  technically  correct,  but  very  different  from  his, 
and  the  disparities  surprised  him  in  interesting  and  useful  ways.  Specifically,  the  car’s  initial 
velocity  was  modeled  as  negative,  but  the  analyst  knew  that  the  car  had  started  from  rest  — 
a  fact  that  he  had  not  mentioned  in  the  input  information.  This  violation  of  his  intuition  not 
only  drew  this  piece  of  implicit  knowledge  (“v  =  0  at  t  =  0”)  out  into  the  explicit  syntax  of 
the  PRET  call,  but  also  suggested  a  piece  of  physics  that  he  had  not  noticed  until  then  —  there 
was  a  delay  in  the  system.  This  delay  is  clearly  visible  on  the  powerpoint  plots  that  accompany 
this  report;  if  one  adds  a  delay  hypothesis  to  the  PRET  call,  the  program  successfully  models 
that  behavior  as  well.  From  a  cognitive  science  point  of  view,  this  interchange  was  particularly 
interesting,  as  the  discrepancies  helped  the  analyst  refine  his  mental  model  of  the  system. 
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Another  important  feature  of  fret’s  behavior  is  also  visible  on  the  plots.  In  this  application, 
the  engineers  cared  deeply  about  the  position  of  the  car,  but  the  heading  and  the  velocity  v 
did  not  matter  as  much.  In  a  case  like  this,  an  engineer  —  who  wants  the  simplest  possible 
ODE  that  models  the  important  behavior  —  does  not  add  terms  to  the  model  to  account  for 
glitches  on  an  unimportant  state  variable,  such  as  the  bump  in  the  velocity  data,  fret  behaves 
in  exactly  the  same  fashion:  it  is  an  engineer’s  modeling  tool,  not  a  scientific  discovery  system; 
because  of  this,  it  is  specifically  designed  to  identify  and  ignore  unimportant  details. 
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